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Over 20 years ago, Robert J. Barak and Barbara E. Breier suggested incorporating a regular assessment of the entire 
program review system into the review schedule in order to ensure that the system itself is as efficient and effective 
as the programs under review. Barak and Breier’ s seminal book on the goals and processes of program review has 
widely influenced the standards for reviews; however, their proposed “meta-review” has not yet become a regular 
element in most higher education institutions. Results from a meta-review undertaken at a small and private liberal 
arts college demonstrate the utility of such an institutional-level assessment. The results reveal that the 
quality — and even the purpose — of program review can shift over time, and that review policies and guidelines 
need to be revised regularly to remain current with emerging external expectations and changing financial 
environments. 

Keywords: program review, assessment, institutional effectiveness, liberal arts 

Program review is one of the most widely used management processes in education. The process typically 
involves self-analysis, peer review, long-range planning, and action research on student learning. Program 
review provides curricular, co-curricular, and academic support units an opportunity to perform a small-scale 
planning exercise specific to their programs. The process is labor intensive and involves numerous, faculty, staff, 
and students. It also requires extensive logistical facilitation as well as a considerable budget to support site visits 
by external consultants. In order to ensure that the program review system itself is as efficient and effective as the 
programs under review, more than 20 years ago, Robert J. Barak and Barbara E. Breier (1990, p. 66) suggested 
incorporating a regular assessment of the system into the overall review schedule. Barak and Breier’s seminal 
book on the goals and processes of program review has influenced the standards for reviews; however, their 
proposed “meta-review” has not yet become a regular element in most higher education institutions. A case study 
of meta-review undertaken at a small, private, liberal arts college demonstrates the utility of such an 
institutional-level assessment. Results reveal that the quality — and even the purpose — of program review can shift 
over time, and that review policies and guidelines need to be revised regularly to remain current with emerging 
external expectations and changing financial environments. The first step that an institution must take in designing 
a meta-review study is to revisit the purpose for which it is making use of the process. 

Program Review in Perspective 

The origin of program review varies considerably depending on how it is defined. Some scholars have 
identified early models for program review in K-12 and specialized accreditation in the 19th century (Black & 
Kline, 2002, p. 224), while others have even pointed to civil service examinations in ancient China as far back 
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as 2000 B. C. (Worther, 1990, p. 42). Still others locate the beginnings of program review in the 17th century 
with Harvard College’s curricular reform of 1642 (Conrad & Wilson, 1985, p. 1). Regardless of its beginnings, 
it is clear that program review has been propelled by the need in higher education for a common tool to assess 
educational quality. Driving factors have included the increased demand for student options (such as the move 
away from Greek and Latin entrance requirements, and the rise of the elective system), and demands for 
operational efficiency (such as state boards seeking to reduce redundancy, and accrediting bodies seeking to 
safeguard federal financial assistance). An extended study commissioned by the National Center for Higher 
Education revealed that although only 12% of post-secondary institutions had initiated a formal program review 
process before the Higher Education Act of 1965 (which established as a financial aid), 43% had done so by 
1975, and as much as 82% had done so by 1982. 

The methodology of program review also began to shift as implementation increased, with qualitative 
approaches beginning to be embraced in the 1970s (Worthen, 1990, p. 43). Early reviews relied heavily on 
quantitative approaches borrowed from budget and planning rooted in the 20th century scientific management 
techniques. It could even be argued that it was the acceptance of action research into the program review 
process that triggered the assessment movement of the 1980s. Outcomes-based learning assessment, of course, 
has its own history within education, but according to Ewell (2002, p. 5), the techniques of early program 
review “provided assessment with a ready-made set of models and vocabularies”. Given this close association 
between program review and assessment, it is intriguing that a lull integration of direct learning studies into 
program review has been slow. Indeed, for many institutions, program review continues to rely on what Sandy 
Astin famously termed the “reputational view of excellence”, e.g., inputs like student selectivity, faculty quality, 
etc. (Astin, 1991, p. 6). Although curriculum relevance and student satisfaction are now fairly standard 
components or program review, in many instances, programs continue to develop their self-studies based on 
resource needs without providing a context of educational effectiveness. 

Full implementation requires institutions to develop a working definition of program review that takes into 
consideration the motivation, activities, essential criteria, and ultimate purpose of its use. Program review has 
the potential to invite confusion with processes that are either conducted within the program review process, or 
that might ran parallel with it. Program evaluation, program approval, program assessment, and program 
prioritization could be considered distinct from program review, yet they also need to be included in the 
discussion. If one was to create a conceptual hierarchy of these various approaches, program evaluation would 
be found at the top. This is the oldest, and in many ways, the most entrenched process. Program evaluation can 
technically include nearly any activity that works towards a value judgment of a given program in order to 
improve the overall system (Worthen, 1990, p. 42). Yet historically, such an evaluation often amounts to a 
cost-benefit analysis, and does not include student learning assessment and curriculum analysis. 

Without respect to methodology, Barak (1982, p. 93) suggested two types of program evaluation based on 
their function. Barak referred to the process of evaluating new programs as program approval, and the process 
of evaluating existing programs as program review. Others have taken another approach, however, stating that 
program evaluation and review are roughly synonymous. Black and Kline (2002, p. 223), for instance, 
explained that “program review” is simply the term used for program evaluation in higher education, with the 
term “program evaluation” used mainly in primary and secondary education. Program-level learning 
assessment has sometimes been put forth as a critical aspect of the definition of program review, but assessment 
is also a suggested criterion for program evaluation of the type recommended by Dickenson (2010, p. 56) in his 
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program prioritization approach. The simple inclusion of assessment, therefore, does not distinguish program 
evaluation form program review. 

Taken together, it seems that the methods involved are less important than the overall aim of these 
processes, but it should be kept in mind that methods can also facilitate a purpose even when unintended. From 
another perspective, it is equally tempting to align program evaluation and program review, with institutional 
accountability and student learning improvement respectively 1 . Although the accountability-improvement 
dichotomy is somewhat unstable (since improvement is the “object” of accountability), the potential inequities 
of who is held accountable are very real. Program evaluation (approval and review, “a la” Barak) in state-wide, 
mutli-institution educational systems, for example, is an inescapable process closely overseen by external 
observers. In contrast, private colleges perfonn program review with substantially less transparency (at least for 
the moment). Accrediting bodies set expectations for all institutional types, but it might be said that some 
institutional types have more flexibility in their application of it. Nevertheless, to say that program evaluation 
serves external accountability demands while program review serves internal interests in improvement is highly 
misleading. Such an alignment addresses the motivation for the review without attempting to determine if 
differences exist in their overall purpose. 

Clearly both accountability and improvement factor into the program evaluation/review mix, as can be 
seen in the writings of many leading education scholars (Suskie, 2009, p. 14; Barak & Mets, 1995, p. 19; 
Conrad & Wilson, 1985, p. iv), and quality assurance is perhaps the most fundamental goal of both program 
evaluation and review. “Beyond cost-saving considerations of academic program evaluation”, wrote Barak, 
“lies a deeper concern for ensuring the inherent quality of our postsecondary institutions” (Barak, 1982, p. 5). 
In a recent book focused on integrating assessment into program review, furthermore, Bresciani stated that the 
primary purpose of program review is to “engage in a systematic, reflective process” that allows for data 
collection and program improvement (Bresciani, 2006, p. 63). 

Bresciani’s (2006) definition emphasizes the act of planning as a key aspect of the purpose of program 
review, and this provides a new vantage point. Planning in her model, is done primarily by the practitioners 
themselves, and the final action plan put forward as a result of the planning is developed by those 
practitioners. By insisting that program review must be based on the program’s stated learning outcomes, 
Bresciani (2006) attempted to ensure not only the self-reflection that she considers essential to the task, but 
also that reviews will be criterion references rather than norm referenced. In other words, programs are 
evaluated against their stated goals and outcomes, not against one another. This is in direct opposition with 
the concept of Dickeson’s (2010) program prioritization, where programs are ranked in order to optimize 
resource allocation. For the purpose that Bresciani (2006) has in mind, criterion references allows institutions 
to remain aligned with their unique mission, and provides programs the flexibility necessary to encourage 
innovation. These structural differences move the purpose of program review in a new direction. Defining 
program review in this way does not preclude using the results for program approval, or resource 
optimization, but it does not limit itself to those uses either. 

Both Bresciani (2006) and Barak (1982) also highlighted another complication with undertaking program 
evaluation of any kind; specifically, what is a program? Historically in higher education, program most 
typically refers to a degree program. Even in the second edition of Linda Suskie’s influential work Assessing 

1 Peter Ewell has discussed the tensions between the use of outcomes-based assessment for both external (and internal) 
accountability as well as for the purpose of improving student learning (Ewell, 2008; 2009). 
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Student Learning, program review is described as a “comprehensive evaluation of an academic program” 
(Suskie 2009, p. 14). Barak (1982) made note of this issue as early as 1982, but did not vehemently argue for 
including other types of programs in program review at the time. Following the lead of recent work by a 
number of national associations that have recognized the important role of student affairs in student learning, 
Bresciani includes co-curricular programs in her program review model as well. This opens up new possibilities, 
but institutions may detennine an even broader conception. Dickeson, for example, recommended that a 
program can be defined as any activity that requires institutional resources (Dickeson, 2010, p. 56). Worthen 
would have likely tempered Dickeson’s definition, as he suggested that a program should be directed at 
addressing a particular educational problem or need (Worthen, 1990, p. 42). Practically speaking, at many 
institutions, a full implementation of program review will be limited to the amount of available resources to 
support it, yet this need not to influence what the institution formally identifies as a reviewable unit. 

Finally, the criteria covered in program review will be determined to a great extent by its purpose, and it is 
precisely for this reason that a clear definition is imperative. It is as simple as having the right tool for the job, 
and in turn, the right data for the analysis. Applying misaligned criteria will lead the analysis to the wrong 
direction regardless of what the process is called. This sort of misdirected analysis can also result in a subtle 
emphasis on one or more criteria over others. If the intent is to improve student retention, for example, a 
substantial analysis of existing faculty office space would not likely provide much useful information. If the 
intent is to improve faculty retention, however, the same analysis might provide critical information. It is due to 
the need for this type of flexibility that a standard set of criteria has not emerged from the literature, although 
suggestions have been made for guiding principles (Bresciani, 2006, pp. 63~96; Barak, 1982, pp. 66-69). 
Conrad and Wilson’s 1985 Association for the Study of Higher Education report identified four primary models 
of program review: goal-based model, responsive (to a pressing issue) model, decision-making model, and 
comioisseur/peer review model (Conrad & Wilson, 1985, p. iv). Although in most cases, program review is 
undertaken with a combination of most or all of these elements, the emphasis on one or the other can be 
manipulated. For instance, program review for accreditation might rely heavily on the goal-based and peer 
review models, while program prioritization resulting from budget cuts might rely on responsive and 
decision-making models. 

If it is not already apparent, program review and program prioritization have become established as 
discrete processes more than any other that has been discussed (see Figure 1). Both processes have ostensibly 
developed out of program evaluation. One might wish to retain the concept of program evaluation as the 
moment of judgment and decision resulting from either a program review or prioritization exercise. In addition, 
program approval might be conceived of as synonymous with program review (although perhaps with modified 
criteria), since the program must be in place in order to undergo the analysis. If the program is not in place, the 
process would be a program proposal. Program assessment, as has been shown, can be a supporting activity of 
either a review or a prioritization. 

The primary fissure between prioritization and review lies not in the criteria or the activities involved. The 
difference lies both in their overall purposes, and their uses, which as previously said has an effect on the 
criteria and activities. Without a doubt, the motivation behind any given process might easily determine the 
purpose, and thus, lead an institution’s hand in deciding which process is to undergo. This intertwining of 
motivation, purpose, use, activities, and criteria must therefore be resolved before embarking on one or the 
other process to ensure that it is managed appropriately. 
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Both prioritization and review employ the best practices and guiding principles of any collaborative 
planning exercise (clear expectations, objectivity, comprehensiveness, use of multiple perspectives and 
methods for triangulation, goal setting, and so on), and they both seek to evaluate quality at the program level. 
The two processes should be thought of as equally valid options, but they do have separate ends. The purpose 
of program review is to evaluate programs against the institution’s mission and strategic priorities, and against 
comparable programs at other institutions. The purpose of program prioritization, in contrast, is to evaluate 
programs against the institution’s mission and strategic priorities, and against other programs at the same 
institution. The main objective of program review is to verify a program’s quality in order to approve, accredit, 
or simply improve upon it. The main objective of program prioritization is to verify a program’s quality in 
order to make resource decisions, or possibly even terminate the program. Program prioritization’s key benefit 
is that it directly links the evaluation to the budget and planning. It is a link that one might call the Achilles’ 
heal of program review (Barak & Mets, 1995, p. 32; Dickeson, 2010, p. 60). Another advantage of 
prioritization is that all programs are done simultaneously, rather than scheduled over multiple years. Even a 
small college can spend over 10 years on one cycle of program reviews. 

Concurrent evaluation speeds up the process and saves the costs of hosting site visits, but it also 
significantly limits a program’s ability to develop a substantive self-study and eliminates the site visit 
(otherwise known as peer review). Peer review is a process used in publication and accreditation, and is one 
that faculty understand and respect. In addition, many institutions will find it challenging to identify a single set 
of criteria for prioritization, or even to form a single committee to make the recommendations that will have 
enough legitimacy to satisfy the entire campus. Prioritization therefore brings it with the risk of being received 
with suspicion by faculty and staff. Those most closely associated with a given program, for example, would 
not be in the position to rank themselves against other programs. Consequently, they might feel alienated from 
a process that by its own design limits the opportunities for full transparency. Still, the aim of the present essay 
is not to debate the merits of either prioritization or review. Program prioritization is presented here as a viable 
alternative to program review in order to make the point that institutions need to consider these two types of 
program evaluation with care. In the case study that follows, program review has been determined as the 
process that best meets the needs of the college. Program review, in this context, is defined as follows: “A 
program-level, integrated planning process performed regularly by any annually-funded and publically-visible 
activity that facilitates student learning”. 
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Since, as Allen put it, the bottom line for education is the generation of learning (Allen, 2004, p. 19), one 
might suggest that every function on campus facilitates student learning in some way. Notwithstanding this 
suggestion, in practice, boundaries for undertaking program review are often dictated by resource constraints. 
The approach taken in the following case study is to place programs into three groups: academic (both degree 
and non-degree), academic support (peer tutoring/mentoring, advising, etc.), and co-curricular (living-learning 
communities, student government, etc.). It is recognized that each of these program types contribute uniquely to 
student learning, and therefore, should be reviewed. No formal program review system currently existed at the 
case-study college for academic support and co-curricular programs, so the focus was on the degree-granting 
programs. Yet, an assessment of the achievements of academic program review is useful not only for academic 
programs, but also for the entire college. After reaffirming the purpose of program review, the case-study 
college was able to design and implement a meta-review assessment of academic program review, and put the 
results to be used for improving the system and creating a model for the rest of the college. 

Meta-review Case Study 

LibArts College is a small, private, residential, and liberal arts college located in an urban setting. The 
college enrolls approximately 2,000 undergraduate students and offers degrees in 29 majors. A fonnal 
academic program review process was established at LibArts in 2001, partly in response to accreditation 
standards. According to several faculty who participated in the process, however, the need for a formal method 
of evaluating program quality had been stressed by faculty for some time. Program review was implemented by 
a faculty-led committee that began the effort by developing a set of guidelines for the process. The Program 
Review Committee was disbanded a few years later, and their tasks were taken up by the Dean’s Office. The 
guidelines were substantially revised in 2006, and again in 2009, with the emphasis on outcomes assessment in 
flux. The 201 1 program review guidelines now have been incorporated together with the assessment guidelines 
in the college’s Assessment Handbook. 

Each academic program has an assessment plan that presents its mission, goals, learning outcomes, 
curriculum map, and assessment schedule. By the initial projections made in 2001, all of the academic 
programs were scheduled to be complete by 2010. This goal was accomplished for all but two of the 29 
programs. Many departments have also developed their own rubrics, surveys, portfolios, and other tools. 
During the interim years between external reviews, every academic department is expected to submit an annual 
assessment report on the year’s assessment. This report typically includes the outcomes assessed, methods used, 
summary results, and any action taken or proposed. The annual report is a recently developed expectation, so in 
most cases, only 1-2 annual assessment reports have been developed by each department. Direct outcomes 
assessment was included in the initial program review guidelines in 2001, yet a mechanism to track annual 
assessment was not introduced until 2009. In addition, a mechanism to approve revisions to existing 
departmental assessment plans has never existed. Specifically, there has been no means to ensure that 
curriculum maps are complete and up to date beyond their initial approval by the Academic Planning 
Committee. 

Finally, although assessment activity has been ongoing since at least 2001, the quality of the assessment 
studies had never been formally reviewed. Program rubrics have sometimes not been based explicitly on 
learning outcomes, for example, the methodology for data collection and interpretation was infrequently 
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questioned, the use of assessment results was left entirely to the discretion of the department, and so on. 
Considering the length of time program review had been functioning without scrutiny, a study was initiated by 
the Dean’s Office in 2010 at the close of the first program review cycle. Using the concept of meta-review laid 
out by Barak and Breier, a team including the vice president for academic affairs, three associate deans, and the 
assessment director undertook a project to assess the system of program review. The intent of the meta-review 
was to review, realign, and reenergize the program review system at the critical transfer from the first cycle to 
the second. 


Methodology 

The meta-review was guided by three primary questions about the existing program review process; 
namely, how effectively is the current program review system meeting the needs of faculty: (1) as a tool to 
enhance student learning? (2) as a tool to enhance the overall student experience? and (3) as a tool to support 
program and institutional planning? 

The data used to address the questions came from multiple sources. The first phase of the study involved a 
direct assessment of program review documentation. A random sample of five programs was identified that 
ensured representation from each academic division (arts and humanities, social sciences, and sciences). The 
documentation assessed included the program’s most current assessment plan, annual assessment report, 
program review self-study, and external consultant report accompanying the most recent program review. The 
assessment team developed an analytic rubric for each document type informed by the college’s Assessment 
Handbook, as well as Western Association for Schools and Colleges (Accrediting Commission for Senior 
Colleges and Universities) rubrics for Program Learning Outcomes and Program Review, recent presentations 
on program review made by Trudy Banta' and Cyd Jenefsky , and related literature on best practices. A 
lengthy assessment session was held where the team scored the documentation as a group with a discussion 
following each documentation type. 

Each of the 20 documents reviewed in the direct assessment were scored according to 1 6 criteria. For the 
Assessment Plan, criteria addressed the quality of the program’s mission, learning outcomes, curriculum map, 
and implementation plan. With the Assessment Report, the criteria covered the presentation of findings, 
methodology, collaboration, and use of the findings. The criteria for the Self-study included the learning goal 
achievement, curricular review, overall student experience, and resource planning. And for the External Report, 
the criteria focused on the use of evidence, alignment with the self-study, relevance for the program, and 
realistic recommendations. The criteria in each rubric was scaled according to whether the elements were “well 
established”, “developing”, “underdeveloped”, or “not found in the document” (see Appendix A). 

The second phase of the study involved first-person experiences, documented evidence, and survey results. 
A focus group session was held that included department chairs and other program representatives of the five 
randomly selected departments in the direct assessment. A structured interview approach was taken that 
addressed the three primary questions guiding the meta-review. Unrelated to the meta-review, all department 
chairs were asked to complete Curricular Planning Reports to prepare for the college’s strategic planning 
process. The first question on the report template inquired about the effectiveness of program review for each 


2 Trudy Banta, institute session. Program Review, WASC Assessment Leadership Academy Session, 2010. 

3 Cyd Jenefsky, institute session, Assessment and Program Review, WASC Assessment Retreat II, 2009. 
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department, and the responses were analyzed for the purpose of the assessment. In addition, local questions 
addressing program review and assessment were added to LibArts’s 2011 administration of the Higher 
Education Research Institute’s Faculty Survey. And finally, a literature review was conducted to gather ideas 
regarding the use and purpose of program review. 

Results of the Documentation Assessment 

The standard of quality for all documentation that was agreed upon by the team was as follows in Table 1 : 
At least 80% of all 16 criteria will be “well established” or “developing”, and at least 60% all criteria will be 
“well established” (see Table 1). 

Table 1 


Quality Standards for All Documentation 


Documents 

Criteria 

Well established 

Assessment Plan 

Mission 

Provides a concise description of the program’s overall purpose and function, and is 
closely aligned with the mission of the college 

Learning outcomes 

Clearly explain behaviors that demonstrate mission-aligned knowledge, skills, and 
attitudes to be developed within the program’s courses 

Curriculum map 

Aligns all learning outcomes with all appropriate courses according to developmental 
levels 

Implementation 

Includes an assessment schedule, potential assessment tools, a plan for sharing and 
maintaining results, and responsibility assignments 

Assessment Report 

Presentation of 

findings 

Clearly explains what was learned and its relevance to the program with references to 
specific evidence 

Methodology 

Describes the focus of the assessment, the tools that were used, who was involved, and 
the evaluation process 

Collaboration and 

sharing 

Describes how data was collected and results were discussed with all program and other 
appropriate faculty, staff, and shidents 

Use of findings 

Explains all actions taken and/or planed, and how the results will be incorporated into 
program and/or college planning and budgeting 

Self-study 

Goal achievement 

Investigates student achievement of all program learning outcomes, and the program’s 
contribution to the institutional mission and commitments 

Curriculum review 

Investigates the relevance of the curriculum with respect to professional and/or academic 
practice, and peer programs at other institutions 

Student experience 

Investigates student satisfaction with the program, retention and graduation rates, and the 
long-term impact of the program 

Resource planning 

Investigates current resources, justifies needs for the achievement of the program’s stated 
goals, and project’s future needs for the next 3-5 years 

External Report 

Use of evidence 

Relies extensively on the data provided in the self-study and during the site visit, and 
interprets it accurately to justify recommendations 

Alignment with 
self-study 

Directly responds to the program’s self-study, including goal achievement, curriculum, 
student experience, and resources. 

Relevance for 

program 

Addresses all of the major issues raised in the self-study, as well as relevant issues in the 
professional and/or academic practice of the discipline 

Realistic and 

actionable 

Makes appropriate recommendations for the college, and provides reasonable suggestions 
for moving them forward 


In the assessment, 1 1 of the 1 6 measures met this standard, with the lowest scores found in the self-study 
and external review report criteria. Assessment Plans scored the highest against the criteria for quality learning 
outcomes, mission alignment, thorough curriculum mapping and well-conceived implementation. The most 
common score in each case was “well established”, and the average score fell somewhere between 
“developing” and “well established”. 

Assessment Reports also performed well against the standard in criteria for presentation of findings, sound 
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methodology, collaboration and sharing, and demonstrated use of findings. All criteria met the standard, 
however, averaging the scores revealed that the use of findings and collaboration and sharing did so minimally. 

The scores were lower overall for the Self-studies and the External Reports, and they were also more wide 
spread between criteria. The most common score for the criterion of resource planning was “well established”, 
for example, while the most common score for the consideration of student experience was “not found in the 
document”. Most importantly, scores for presentations of student experience, goal achievement, and curriculum 
relevance in the self-study all fell short of the expected standard. 

Somewhat surprisingly, External Reports scored higher than institutional Self-studies. The External 
Reports met the standard, although minimally, in the criteria of relevance for the program, and in alignment 
with the Self-study. The external reports did not meet the standard for scores in the criterion for providing a 
realistic and actionable plan for programs, with the most common score being “underdeveloped”. The use of 
evidence criterion in the external reports was also found to be “underdeveloped” and scored the lowest against 
the standard. 


Results of Curricular Planning, Focus Group, and Faculty Survey 

During the years of 2009-2010, all academic departments were asked to prepare a curriculum planning 
report for the new college president, as well as the incoming vice president for academic affairs. The report was 
intended to serve several purposes, but was primarily aimed at orienting the new president and new dean to 
academic departments, and as a preparatory exercise for a planning process and a capital campaign. The first 
prompt departments were to respond to in the report was: Please begin by offering a response to your most 
recent external review. What did they get right? What did they miss? 

The effectiveness of program review was demonstrated in nearly all of the reports submitted. Departments 
overwhelmingly described its usefulness, and many provided a status report on their progress implementing the 
resulting recommendations. For examples, this perspective can be seen in the words of department chairs 
themselves: 

As a department, we agree completely with our most recent external review. Their suggestions were based on 
accurate facts, were professionally thought out, and were caringly worded. 

This review provided an important context for department discussions and eventual major curricular revisions over 
the subsequent three years. . . 

Overall, we believe that the external reviewers recognized the many strengths of the department and made excellent 
recommendations that have helped us strengthen our curriculum and major. 

There were some issues raised, however, particularly regarding follow-up after the review had taken place. 
This issue also surfaced during the focus group discussion. The focus group was primarily guided by the main 
research questions for the meta-review, but based on the indication of a problem with follow-up in the planning 
reports, the issue was raised again with the smaller group. Focus group participants reaffirmed the effectiveness 
of program review overall, stating that before the review system, the existed program evaluations were trivial 
and without any external validation. Yet, the group also confirmed that follow-up after the review had been 
spotty. They also expressed problems with balancing workload, and the overall coordination of the process. 
One department complained about never receiving feedback of an action plan from the Dean’s Office, while 
another stated that they were never made aware that the external report had come in until months after the fact. 

The focus group consisted of mostly department chairs, and the curricular planning reports were written by 
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department chairs (although with departmental faculty input). Since it is expected that all faulty participate in 
program review to some extent, a faculty survey was used to collect further evidence. Faculty were asked two 
sets of questions, the first being based on the research questions for meta-review. In this instance, the questions 
asked if faculty agree that “Our program review process is meeting my needs” as a tool to enhance program and 
institutional planning, student learning, and the overall student experience. The response from the faulty as a 
whole was not as positive as the response from the department chairs. Based on the results with a response rate 
of 48%, the percentage of faculty who strongly agreed with these questions did not exceed 10% for any of them. 
For the total in agreement (“Agree” + “Strongly agree”), the average between the three questions was 
approximately 35% of the sample (see Figure 2). 



Figure 2. Percent of criteria receiving scores of “well established” or “developing”. 


The second set of questions focused on the perceived value of assessment, which has implications for the 
linkage between assessment and program review (see Figure 3). A hypothesis was made that if faculty believed 
that the student learning outcomes for their program are the right outcomes, and that the curriculum for their 
program offers adequate opportunities to achieve the learning, and finally that the students in their program are 
in fact achieving those learning outcomes, that they, as faculty, would find value in this process regardless of 
whether they found any personal satisfaction in it. Interestingly, while an average of 80% of the faculty 
responded in agreement to the first three of these questions, only 54% agreed that the process of assessment 
was valuable. The response is difficult to concisely interpret without more analysis, but ostensibly there is a 


750 


META-REVIEW: SYSTEMATIC ASSESSMENT OF PROGRAM REVIEW 


misalignment between assessment’s functional and perceived value (see Figure 4). While this predicament 
might not be unusual among faculty, it should not be overlooked. 



Figure 3. The percentage of faculty’s agreement with that “Our program review process is meeting my needs”. 



Figure 4. The percentage of faculty’s agreement with questions focus on the “Perceived value of assessment”. 

Discussion 

Considering the results of the documentation assessment, and the feedback gained from the planning 
reports, focus group, and faculty survey, the assessment team found much evidence upon which to base both a 
thorough understanding of the current status of program review, and the recommendations needed to move it 
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forward. It is clear that program review was functioning to good effect in support of student learning, student 
satisfaction, and planning, but it was also clear that there are numerous loose ends unraveling that need to be 
re-woven together. 

The assessment team did not take a judgmental or comparative approach to the analysis due to their 
understanding that any process has the potential to drift from its initial objectives over time. Assumptions about 
why one or another criterion did not meet the standard were not made, and no conclusion about a given 
department’s commitment to assessment was fonnulated. It was noted in the discussion that followed the 
assessment that not only have LibArts’s program review and assessment guidelines been revised since the 
earliest reviews took place, but also accreditation expectations and specificity about best practices have become 
more explicit as well. In other words, the rubrics developed were based on present expectations, which may or 
may not have been well understood by a department that went through the process several years ago. 

The aim of the assessment was not to find flaws within departmental practices, but to evaluate how well 
departments are meeting current expectations. Yet, interestingly enough, departments that were reviewed 
earlier in the cycle did not necessarily perfonn more poorly than those that went later. In fact, the assessment 
indicated that the quality of the documentation was in some cases higher in past years than in more recent years. 
Due to the limits of the study, a specific reason for this could not be determined, but it was speculated that 
departments may have been more engaged in the process at the beginning of the cycle. A more progressive 
approach to the assessment with a higher standard of quality might assume that the college would keep pace 
with internal and external changes. And further, after 10 years of institutional learning, it is not unreasonable to 
expect that departments would have increased their facility with performing program review. If the criteria 
were measured against a higher standard of 80% of the criteria receiving scores of “well established”, for 
example, the results would be disappointing. This underscores the importance of predetennining just how 
“good” is good enough. 

Similar to defining the purpose and scope of program review, establishing a standard of quality for what is 
expected is a critical discussion to undertake before beginning a meta-review. The same might be said for any 
assessment, since without some kind of benchmark, assessment can be somewhat meaningless. The scores 
themselves have no value unless compared with something. In many ways, it is the act of comparing that 
facilitates interpretation, and more importantly, inspires reflection and discussion around the findings. The 
results of the meta-review in the case study above included a healthy list of recommendations for improving the 
efficiency and effectiveness of program review, but as with any assessment the real value was in the subsequent 
faculty and staff deliberations that took place on the campus. 

One key finding that was not easily resolved in the recommendations for improvement underscores a 
fundamental problem with program review. It revolves around an issue which program prioritization attempts 
to resolve: the link between process and budget. Indeed, there are many concerning issues that might potentially 
arise in the post-review stage of the process. A typical problem is that momentum often dissipates after the site 
visit, and by the time the external report arrives, it receives little attention. The process comes to a halt, and 
faculty are left wondering why they spent so much energy for so little return. A study in 1995 by Barak and 
Mets found that one of the drivers of this phenomenon is that recommendations from the report often fall into 
three categories: those for the department, those for the administration, and those that seeming fall somewhere 
in between (Barak & Mets, 1995). The first problem with this is that some of the recommendations, and in 
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Barak and Mets’ study as much as 30%, do not have a clear owner. Equally problematic is that the remaining 
recommendations are divided between faculty and administration, setting the stage for a stalemate if one or the 
other does not respond. Perhaps more difficult is that program review recommendations usually require 
resources to implement — sometimes substantial resources — that might not exist. 

The logical response to this problem would be to set aside a specific annual budget for program review 
implementation. This can be a hard sell, but here is another way that understanding program review as a 
planning exercise can be helpful. Institutions must set aside funds for strategic planning, otherwise, the efforts 
of the campus will go unrealized. The same should be true of program review, yet on a smaller scale of course. 
If program review is going to be effective it must be supported. On the other hand, if the purpose of program 
review is to improve the quality of our programs, and the bottom line for education is learning, programs can 
do a lot themselves simply by focusing on student learning instead of on departmental resource needs. The 
solution, then, requires collaboration. Administrations should attract or reallocate funds to realize needed 
improvements discovered in program review, and academic departments should focus their self-studies on what 
they most closely control (e.g., their curriculum). Such collaboration will breathe new life into program review 
and reassert its usefulness and value for higher education. 
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Appendix A 


Table A 1 

Assessment Plan Rubric 


Criterion 

Not found 

Underdeveloped 

Developing 

Well established 

Mission 

1 — 1 missing 

from the 

document 

L_i provides an 

incomplete description of 
the program’s purpose 
and function 

1 — 1 provides a description 
of program’s purpose and 
function and links to the 
mission of the college 

1 — 1 provides a concise description 
of the program’s overall purpose 
and function, and is closely aligned 
with the mission of the college 
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(Table A1 continued) 


Learning outcomes 

1 — 1 missing 
from the 
document 

L! explain the 

knowledge, skills, and 
attitudes to be 

developed within the 
program’s courses 

i — 1 explain student 

behaviors that demonstrate 
the knowledge and/or skills 
and/or attitudes to be 
developed within the 

program’s courses 

Lfc clearly explain student 
behaviors that demonstrate 

mission-aligned knowledge, skills, 
and attitudes to be developed within 
the program’s courses 

Curriculum map 

| | missing 

from the 

document 

Li aligns most 

learning outcomes with 
courses 

1 — 1 aligns all learning 

outcomes with appropriate 
courses according to 

developmental levels 

1 — 1 aligns all learning outcomes 
with all appropriate courses 
according to developmental levels 

Implementation 

! — ■ missing 

from the 

document 

1 — 1 includes an 

assessment schedule 

1 — 1 includes an assessment 
schedule, potential 

assessment tools, and a plan 
for sharing and/or 

maintaining results 

1 — 1 includes an assessment 

schedule, potential assessment 
tools, a plan for sharing and 
maintaining results, and 

responsibility assignments 


Comments 


Table A2 


Assessment Report Rubric 


Criterion 

Not found 

Underdeveloped 

Developing 

Well established 

Presentation of 
findings 

1 — 1 missing from the 
document 

1 — 1 explains what 
was learned 

1 — 1 explains what was learned 
an its relevance to the program 

LL clearly explains what 
was learned and its 
relevance to the program 
with references to specific 
evidence 

Methodology 

1 — 1 missing from the 
document 

1 — 1 describes the 
focus of the 

assessment 

1 — 1 describes the focus of the 
assessment, the tools that were 
used and/or who was involved 

1 — 1 describes the focus of 
the assessment, the tools that 
were used, who was 
involved, and the evaluation 
process 

Collaboration and 

sharing 

1 — 1 missing from the 
document 

1 — 1 describes how 
data was collected 
and/or results were 
discussed with other 
program faculty 

1 — 1 describes how data was 
collected and/or results were 
discussed with all program 
faculty and/or staff 

1 — 1 describes how data was 
collected and results were 
discussed with all program 
and other appropriate 

faculty, staff, and students 

Use of findings 

1 — 1 missing from the 
document 

1 — J explains actions 
taken and/or 

planned 

1 — 1 explains actions taken 
and/or planned, and how results 
might be incorporated into 
program planning and/or 

budgeting 

1 — 1 explains all actions 
taken and/or planed, and 
how the results will be 
incorporated into program 
and/or college planning and 
budgeting 


Comments 


Table A3 


Self-study Rubric 


Criterion 

Not found 

Underdeveloped 

Developing 

Well established 

Goal achievement 

1 — 1 missing from 
the document 

1 — 1 investigates student 
achievement of some 
program learning 

outcomes, and/or the 
program’s contribution to 
the institutional mission 

1 — 1 investigates student 

achievement of most program 
learning outcomes, and the 
program’s contribution to the 
institutional mission 

i — ! investigates student 

achievement of all program 
learning outcomes, and the 
program’s contribution to the 
institutional mission and 

commitments 
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(Table A3 continued) 


Curriculum review 

U missing from 
the document 

1 — 1 investigates the 

relevance of the 

curriculum with respect to 
professional and/or 

academic practice 

1 — 1 investigates the relevance 
of the curriculum with respect 
to professional and/or 

academic practice, or peer 
programs at other institutions 

1 — 1 investigates the relevance 
of the curriculum with respect 
to professional and/or 

academic practice, and peer 
programs at other institutions 

Student experience 

1 — 1 missing from 
the document 

1 — 1 investigates student 
satisfaction with the 
program, or retention and 
graduation rates 

1 — 1 investigates student 

satisfaction with the program, 
and retention and graduation 
rates 

1 — 1 investigates student 

satisfaction with the program, 
retention and graduation rates, 
and the long-tenn impact of the 
program 

Resource planning 

1 — 1 missing from 
the document 

Lt investigates current 
resources and states the 
program’s needs 

1 — 1 investigates current 

resources, and justifies needs 
for the achievement of the 
program’s stated goals 

1 — j investigates current 

resources, justifies needs for 
the achievement of the 
program’s stated goals, and 
project’s future needs for the 
next 3—5 years 


Comments 


Table A4 

External Report Rubric 


Criterion 

Not found 

Underdeveloped 

Developing 

Well established 

Use of evidence 

1 — 1 missing from the 
document 

1 — 1 report relies on 

some of the data 

provided in the self 
study and/or during the 

site visit to make 

recommendations 

1 — 1 report relies mostly 
on the data provided in 
the self study and during 
the site visit to make 
recommendations 

!— i report relies 

extensively on the data 
provided in the self 
study and during the site 
visit, and interprets it 
accurately to justify 
recommendations 

Alignment with self study 

1 — 1 missing from the 
document 

1 — 1 report responds 
minimally to the 

program’s self study, 

including goal 

achievement, 
curriculum, student 

experience, and/or 

resources 

1 — 1 report responds 
somewhat to the 

program’s self study, 
including goal 

achievement, 
curriculum, student 

experience, and/or 

resources 

1 — 1 report directly 

responds to the 

program’s self study, 

including goal 

achievement, 
curriculum, student 

experience, and 

resources 

Relevance for program 

1 — 1 missing from the 
document 

1 — 1 report addresses 

some of the major issues 
raised in the self study 

1 — 1 report addresses 

most of the major issues 
raised in the self study 

1 — 1 report addresses all 
of the major issues 
raised in the self study, 
as well as relevant issues 
in the professional 

and/or academic practice 
of the discipline 

Realistic and actionable 

1 — 1 missing from the 
document 

I — - report makes 

recommendations that 
are somewhat 

appropriate for the 

college 

! — 1 report makes 

appropriate 

recommendations for the 
college 

i — i report makes 

appropriate 

recommendations for the 
college, and provides 
reasonable suggestions 
for moving them forward 


Comments 





